Papers with knowledge injection
CogKTR: A Knowledge-Enhanced Text Representation Toolkit for Natural Language Understanding (2022.emnlp-demos)
Copied to clipboard
Zhuoran Jin, Tianyi Men, Hongbang Yuan, Yuyang Zhou, Pengfei Cao, Yubo Chen, Zhipeng Xue, Kang Liu, Jun Zhao
| Challenge: | Existing knowledge-enhanced methods are limited to knowledge-intensive tasks. |
| Approach: | They propose a knowledge-enhanced text representation toolkit for natural language understanding . it combines knowledge acquisition, knowledge representation, knowledge injection and knowledge application . |
| Outcome: | The proposed toolkit supports knowledge acquisition, knowledge representation, knowledge injection, and knowledge application. |
Enhancing Self-Attention via Knowledge Fusion: Deriving Sentiment Lexical Attention from Semantic-Polarity Scores (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods to inject lexical features into self-attention mechanisms have shown remarkable performance across various downstream tasks in NLP. |
| Approach: | They propose to inject lexical features into the self-attention mechanism of Transformer-based models by injecting lexicon-based Sentiment Lexical Attention into the attention scores throughout the training process. |
| Outcome: | The proposed method shows significant performance improvements on the NSMC sentiment classification benchmark and is able to perform in out-of-domain tasks. |
Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in language modeling have enabled applications across multiple domains such as education, jurisprudence, and healthcare. |
| Approach: | They propose a method to use knowledge to identify which rare words are important and uplift their conditional probability. |
| Outcome: | The proposed approach reduces the uncertainty of the model and improves factuality and coherence without negatively impacting fluency. |
Does the Correctness of Factual Knowledge Matter for Factual Knowledge-Enhanced Pre-trained Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work neither proves that pre-trained models successfully learn the injected factual knowledge nor proves there is a causal relation between injected knowledge and downstream performance improvements. |
| Approach: | They propose a counterfactual-based analysis framework to explore the causal effects of factual knowledge injection on the performance of language models within pretrain-finetune paradigm. |
| Outcome: | The proposed framework shows that factual knowledge injection is successful but correctness of injected knowledge only has limited effect on the models’ downstream performance. |
Ensemble Privacy Defense for Knowledge-Intensive LLMs against Membership Inference Attacks (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are the foundation of modern natural language processing, powering applications across diverse domains. |
| Approach: | They propose a model-agnostic defense framework which aggregates and evaluates the outputs of a knowledge-injected LLM, a base LLM and a dedicated judge model to enhance resistance against membership inference attacks. |
| Outcome: | The proposed framework reduces MIA success by up to 27.8% for SFT and 526.3% for RAG compared to inference-time baseline while maintaining answer quality. |
Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives (2025.findings-acl)
Copied to clipboard
Zhihu Wang, Shiwan Zhao, Yu Wang, Heyuan Huang, Sitao Xie, Yubo Zhang, Jiaxin Shi, Zhixing Wang, Hongyan Li, Junchi Yan
| Challenge: | Existing approaches to solving complex tasks with large language models (LLMs) fail to decompose tasks accurately or execute subtasks effectively. |
| Approach: | They propose a Chain-of-Learning (CoL) paradigm that highlights task dependencies on specific capability items, further broken down into their constituent knowledge and skill components. |
| Outcome: | The proposed model improves Yi-1.5-9B and Llama3-Chinese-8B for legal tasks by 45.00% and 24.50% on different domains. |
Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for updating knowledge show little propagation of injected knowledge. |
| Approach: | They propose to inject individual facts into LMs and evaluate whether they can propagate injected facts while not changing predictions on other contexts. |
| Outcome: | The proposed model can make inferences based on injected facts and propagate them . existing methods show little propagation of injected knowledge . |
Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to augment pre-trained large language models require extensive computational efforts and massive data volumes, challenging the widespread accessibility of LLM research. |
| Approach: | They propose a post-pretraining strategy of selectively enhancing shallow layers while pruning less effective deep ones to augment pretrained large language models. |
| Outcome: | The proposed approach improves performance on the corpus of code & math and a legal corpus and is widely applicable. |
Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for language model pretraining use limited knowledge graph data for knowledge-intensive tasks. |
| Approach: | They propose to make better use of multilingual annotations and language agnostic properties of KG triples for pretraining LMs. |
| Outcome: | The proposed models show significant performance improvements on a wide range of knowledge-intensive cross-lingual tasks. |
Jellyfish: Instruction-Tuning Local Large Language Models for Data Preprocessing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Until 2021, most efforts were concentrated on one or two specific tasks such as error detection (ED) and data imputation (DI). |
| Approach: | They propose to instruction tune local LLMs as universal DP task solvers that operate on a local, single, and low-priced GPU, ensuring data security and enabling further customization. |
| Outcome: | The proposed models deliver competitiveness and generalizability to unseen tasks while barely compromising the base models’ abilities in NLP tasks. |
Efficient Dynamic Clustering-Based Document Compression for Retrieval-Augmented-Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | RAG implementations face challenges in addressing retrieved noise and redundant content . current RAG methods lack the ability to exploit fine-grained inter-document relationships . |
| Approach: | They propose a retrieval-augmented generation framework that exploits latent inter-document relationships while removing irrelevant information and redundant content. |
| Outcome: | The proposed framework achieves consistent performance improvements on knowledge-QA and hallucination-Detection datasets. |
Data Doping or True Intelligence? Evaluating the Transferability of Injected Knowledge in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study shows that comprehension-intensive fine-tuning tasks retain knowledge longer . however, all models exhibit significant performance drops when applying injected knowledge in broader contexts . |
| Approach: | study: comprehension-intensive fine-tuning tasks achieve higher knowledge retention rates . larger models show improved retention across all task types, study finds . |
| Outcome: | a new study shows that comprehension-intensive fine-tuning tasks retain knowledge better than mapping-oriented tasks despite exposure to identical factual content. |
Plug-and-Play Knowledge Injection for Pre-trained Language Models (2023.acl-long)
Copied to clipboard
Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang, Deming Ye, Chaojun Xiao, Xu Han, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou
| Challenge: | Existing knowledge injection methods are not suitable for enhancing pre-trained language models with external knowledge bases. |
| Approach: | They propose a plug-and-play knowledge injection method where knowledge bases are injected into frozen existing downstream models by a knowledge plugin. |
| Outcome: | The proposed method improves the performance of knowledge injection on knowledge-driven tasks while keeping model parameters frozen. |
EnigmaToM: Improve LLMs’ Theory-of-Mind Reasoning Capabilities with Neural Knowledge Base of Entity States (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing ToM reasoning methods rely excessively on off-the-shelf LLMs, reducing their efficiency and limiting their applicability to high-order ToM. |
| Approach: | They propose a neuro-symbolic framework that integrates a Neural Knowledge Base of Entity States and knowledge injection to enhance ToM reasoning. |
| Outcome: | The proposed framework improves ToM reasoning on ToMi, HiToM, and FANToM benchmarks. |
Injecting Domain Knowledge in Language Models for Task-oriented Dialogue Systems (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models lack domain-specific knowledge that does not naturally occur in pre-training data. |
| Approach: | They propose to inject domain-specific knowledge prior to fine-tuning on TOD tasks by using adapters that can be easily integrated with PLMs. |
| Outcome: | The proposed methods show that they can inject domain-specific knowledge prior to fine-tuning on TOD tasks. |
KEHRL: Learning Knowledge-Enhanced Language Representations with Hierarchical Reinforcement Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | General pre-trained language models (PLMs) leverage relation triples from knowledge graphs (KGs) and integrate external data sources into language models via self-supervised learning. |
| Approach: | They propose to learn Knowledge-Enhanced language representations with Hierarchical Reinforcement Learning (KEHRL) to detect positions for knowledge injection and integrate external knowledge into the model to avoid injecting inaccurate or irrelevant knowledge. |
| Outcome: | The proposed model can detect essential positions in texts for knowledge injection and integrate external knowledge into the model to avoid injecting inaccurate or irrelevant knowledge. |
Knowledge-enhanced Prompt Tuning for Dialogue-based Relation Extraction with Trigger and Label Semantic (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to determine semantic relation between two arguments in dialogues are limited due to the low information density of text. |
| Approach: | They propose a Knowledge-Enhanced Prompt-Tuning method to enhance DRE model by exploiting trigger and label semantics. |
| Outcome: | The proposed method achieves state-of-the-art in F1 and F1c scores on a DialogRE dataset. |
ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis (2023.findings-acl)
Copied to clipboard
Yakun Yu, Mingjun Zhao, Shi-ang Qi, Feiran Sun, Baoxun Wang, Weidong Guo, Xiaoli Wang, Lei Yang, Di Niu
| Challenge: | Existing methods for multimodal sentiment analysis focus on general knowledge, which is inadequate to identify specific sentiments across modalities. |
| Approach: | They propose a method where specific-knowledge representations for each modality can be learned together with general knowledge representations via knowledge injection based on an adapter architecture. |
| Outcome: | The proposed method outperforms all prior methods on three popular benchmarks on multimodal sentiment analysis metrics. |
Learning Knowledge-Enhanced Contextual Language Representations for Domain Natural Language Understanding (2023.emnlp-main)
Copied to clipboard
Taolin Zhang, Ruyao Xu, Chengyu Wang, Zhongjie Duan, Cen Chen, Minghui Qiu, Dawei Cheng, Xiaofeng He, Weining Qian
| Challenge: | Existing methods for pre-training KEPLMs with relational triples are difficult to adapt to close domains due to the lack of sufficient domain graph semantics. |
| Approach: | They propose a Knowledge-enhanced language representation learning framework for various closed domains that captures the implicit graph structure among the entities. |
| Outcome: | The proposed framework outperforms existing methods for pre-training KEPLMs in closed domains significantly. |
How Can Synthetic Data Improve Multilingual Language Model Pretraining? A Data Quality Perspective (2026.acl-long)
Copied to clipboard
| Challenge: | Low-resource languages are a long-tail problem for multilingual LLMs due to limited high-quality training data. |
| Approach: | They propose a method that translates high-quality, knowledge-rich English data into low-resource languages . they propose SynRank, which leverages synthetic data as positive samples to train a classifier . |
| Outcome: | The proposed method matches handcrafted rule-based filtering by human experts and significantly improves knowledge-intensive tasks with less data. |
LegalChainReasoner: Grounding Criminal Judicial Opinion Generation via Structured Legal Chains (2026.acl-long)
Copied to clipboard
| Challenge: | Current legalAI tasks divide sentencing and legal reasoning into two separate tasks, resulting in inconsistency between the reasoning and predictions. |
| Approach: | They propose a new task that generates both legal reasoning and sentencing decisions using a framework that applies structured legal chains to guide the model through comprehensive case assessments. |
| Outcome: | The proposed model outperforms baseline models on real-world, open-source Chinese legal case datasets. |
Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models capture factual knowledge across a wide range of domains, but refining their capabilities on previously seen knowledge remains a challenge. |
| Approach: | They propose a synthetic knowledge ingestion method that leverages fine-grained synthesis and interleaved generation to construct high-quality data representations from raw knowledge sources. |
| Outcome: | The proposed method outperforms baseline methods on question-answering tasks spanning finance, biomedicine, and open-generation domains. |
Memorizing is Not Enough: Deep Knowledge Injection Through Reasoning (2025.acl-long)
Copied to clipboard
Ruoxi Xu, Yunjie Ji, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Yingfei Sun, Xiangang Li, Le Sun
| Challenge: | Existing knowledge injection frameworks focus on knowledge memorization and retrieval, but static nature of large language models leads to outdated information as the real world evolves or when adapting to domain-specific knowledge. |
| Approach: | They propose a four-tier knowledge injection framework that defines the levels of knowledge injection: memorization, retrieval, reasoning, and association. |
| Outcome: | The proposed framework defines the levels of knowledge injection: memorization, retrieval, reasoning, and association. |
Structure-aware Domain Knowledge Injection for Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Structure-aware Continual Pre-Training (SCPT) and Structure-Aware Supervised Fine-Tuning (SSFT) are two-stage strategies for knowledge injection and alignment that reduces the training corpus needs to 5% while achieving 100% of traditional knowledge injection performance. |
| Approach: | They propose a method to efficiently transform foundation Large Language Models into domain specialists by using two-stage strategies: Structure-aware Continual Pre-Training and Structure-Aware Supervised Fine-Tuning. |
| Outcome: | The proposed method significantly reduces the training corpus needs to a mere 5% while achieving 100% of traditional knowledge injection performance. |
GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have demonstrated that supervised fine-tuning and reinforcement learning are effective in integrating knowledge injection with robust generalization. |
| Approach: | They propose a unified post-training framework that addresses intrinsic limitations of supervised fine-tuning and reinforcement learning. |
| Outcome: | The proposed framework surpasses SFT-based methods and yields policies that integrate more smoothly with subsequent RL training. |